Q. Create a single Pod of image httpd:2.4.41-alpine in Namespacedefault . The Pod should be named pod1 and the container should be named pod1-container . This Pod should only be scheduled on a controlplane node , do not add new labels any nodes.
Troubleshooting: i added toleration for controlplane node and also nodeSelector. But the pod is stuck in pending state. Below is my yaml file:
Hi @uzmashafi061,
Please share the output of the kubectl describe so we can provide you the better insight on this.
Btw did you try with nodeName field?
I used your pod manifest with nodeSelector and changed value of kubernetes.io/hostname to match my clusterβs control-plane hostname. Pod is able to run.
25s Warning FailedScheduling pod/pod1 0/3 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/master: }, 2 node(s) didnβt match Podβs node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
Your better insight is already in the messages you posted.
25s Warning FailedScheduling pod/pod1 0/3 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/master: }, 2 node(s) didnβt match Podβs node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
Your manifest only toleratenode-role.kubernetes.io/control-plane:NoSchedule but not node-role.kubernetes.io/master:NoSchedule
Why is the cluster being set up this way? Please ask Kim WΓΌstkamp
As of v1.20 node-role.kubernetes.io/control-plane was introduced, and node-role.kubernetes.io/master is deemed to be deprecated until removal in future.
I have 2 clusters in my demo below,
kind-kke-master cluster has taints node-role.kubernetes.io/master:NoSchedule AND node-role.kubernetes.io/control-plane:NoSchedule on kke-master-control-plane
kind-kke-control-plane cluster only has taint node-role.kubernetes.io/control-plane:NoSchedule on kke-control-plane-control-plane
Hope this is worth a million words of explanation.
sauron @ mordor in on β΅ kind-kke-control-plane () kubernetes-env/kind on π± dev via π pyenv 3.11.5
Ξ» for context in kke-master kke-control-plane ; do k --context kind-"$context" get no ; done
NAME STATUS ROLES AGE VERSION
kke-master-control-plane Ready control-plane 5m42s v1.28.0
kke-master-worker Ready <none> 5m16s v1.28.0
kke-master-worker2 Ready <none> 5m17s v1.28.0
NAME STATUS ROLES AGE VERSION
kke-control-plane-control-plane Ready control-plane 4m13s v1.28.0
kke-control-plane-worker Ready <none> 3m54s v1.28.0
kke-control-plane-worker2 Ready <none> 3m53s v1.28.0
sauron @ mordor in on β΅ kind-kke-master () kubernetes-env/kind on π± dev via π pyenv 3.11.5
Ξ» for context in kke-master kke-control-plane ; do k --context kind-"$context" get no "$context"-control-plane -oyaml | yq '.spec.taints' | less; done
ββββββββ¬βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β STDIN
ββββββββΌβββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
1 β - effect: NoSchedule
2 β key: node-role.kubernetes.io/master
3 β - effect: NoSchedule
4 β key: node-role.kubernetes.io/control-plane
ββββββββ΄βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
ββββββββ¬βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β STDIN
ββββββββΌβββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
1 β - effect: NoSchedule
2 β key: node-role.kubernetes.io/control-plane
ββββββββ΄βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
sauron @ mordor in on β΅ kind-kke-control-plane () kubernetes-env/kind on π± dev via π pyenv 3.11.5
cat <<EOF > /tmp/pod1.yaml # your original manifest
apiVersion: v1
kind: Pod
metadata:
name: pod1
namespace: default
spec:
containers:
- name: pod1-container
image: httpd:2.4.41-alpine
nodeSelector:
kubernetes.io/hostname: cluster1-controlplane
tolerations:
- key: "node-role.kubernetes.io/control-plane"
operator: "Exists"
effect: "NoSchedule"
EOF
sauron @ mordor in on β΅ kind-kke-control-plane () kubernetes-env/kind on π± dev via π pyenv 3.11.5
Ξ» for context in kke-master kke-control-plane ; do context="${context}-control-plane" yq '.spec.nodeSelector."kubernetes.io/hostname" |= env(context)' /tmp/pod1.yaml| k --context "kind-$context" apply -f - ; done
sauron @ mordor in on β΅ kind-kke-control-plane () kubernetes-env/kind on π± dev via π pyenv 3.11.5
Ξ» for context in kke-master kke-control-plane ; do echo "\nContext: $context" ;k --context kind-"$context" get po ; echo ;done
Context: kke-master
NAME READY STATUS RESTARTS AGE
pod1 0/1 Pending 0 11s
Context: kke-control-plane
NAME READY STATUS RESTARTS AGE
pod1 1/1 Running 0 9s
sauron @ mordor in on β΅ kind-kke-control-plane () kubernetes-env/kind on π± dev via π pyenv 3.11.5
Ξ» k --context kind-kke-master describe po pod1 | grep -A4 -i events
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 10s default-scheduler 0/3 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/master: }, 2 node(s) didn't match Pod's node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling..
sauron @ mordor in on β΅ kind-kke-control-plane () kubernetes-env/kind on π± dev via π pyenv 3.11.5
Ξ» yq '.spec.nodeSelector."kubernetes.io/hostname" |= "kke-master-control-plane" | with(.spec.tolerations; select(.[1] += {"key":"node-role.kubernetes.io/master", "operator": "Exists","effect": "NoSchedule"}| .. style="double"))' /tmp/pod1.yaml | tee /tmp/pod1-updated.yaml | k --context kind-kke-master replace -f - --force
pod "pod1" deleted
pod/pod1 replaced
sauron @ mordor in on β΅ kind-kke-control-plane () kubernetes-env/kind on π± dev via π pyenv 3.11.5
Ξ» for context in kke-master kke-control-plane ; do echo "\nContext: $context" ;k --context kind-"$context" get po ; echo ;done
Context: kke-master
NAME READY STATUS RESTARTS AGE
pod1 1/1 Running 0 18s
Context: kke-control-plane
NAME READY STATUS RESTARTS AGE
pod1 1/1 Running 0 19m
sauron @ mordor in on β΅ kind-kke-control-plane () kubernetes-env/kind on π± dev via π pyenv 3.11.5
Ξ» k --context kind-kke-master get po pod1 -oyaml | yq '.spec.tolerations[]'
effect: NoSchedule
key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
key: node-role.kubernetes.io/master
operator: Exists
effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300