Hi Group , Need quick help to understand below behaviour related to coredns
As soon as i scale up the coredns deployment from 1 to 2 replica the DNS resolution doesn’t wotk with new pod . Kindly help to figure out the reason .
**Scenario 1: With 1 POD under coredns Deployment
Observation: DNS lookup works fine when request is routed to coredns pod (10.200.1.31) via service (coredns/10.32.0.10)
[opc@k8master1 ~]$ kubectl exec -i -t dnsutils – nslookup coredns.kube-system
Server: 10.32.0.10
Address: 10.32.0.10#53
Name: coredns.kube-system.svc.cluster.local
Address: 10.32.0.10
[opc@k8master1 ~]$ kubectl exec -i -t dnsutils – nslookup kubernetes
Server: 10.32.0.10
Address: 10.32.0.10#53
Name: kubernetes.default.svc.cluster.local
Address: 10.32.0.1
Scenario 2: With 2 POD under coredns Deployment
Observation: DNS lookup is broken when DNS request is sent to new coredns POD (10.200.0.24) via service (coredns/10.32.0.10)
[opc@k8master1 ~]$ kubectl exec -i -t dnsutils – nslookup coredns.kube-system
;; reply from unexpected source: 10.200.0.24#53, expected 10.32.0.10#53
Complete setup details for reference:
----- Normal setup with 1 POD under delpoyment
[opc@k8master1 ~]$ kubectl get deployment -n=kube-system -o wide
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
coredns 1/1 1 1 23h coredns coredns/coredns:1.11.4 app.kubernetes.io/instance=coredns,app.kubernetes.io/name=coredns,k8s-app=coredns
[opc@k8master1 ~]$ kubectl get svc -n=kube-system -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
coredns ClusterIP 10.32.0.10 <none> 53/UDP,53/TCP 23h app.kubernetes.io/instance=coredns,app.kubernetes.io/name=coredns,k8s-app=coredns
[opc@k8master1 ~]$ kubectl get pod -o wide --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
default dnsutils 1/1 Running 0 23h 10.200.0.17 k8worker1.workersubnet.testvcn.oraclevcn.com <none> <none>
default nginx 1/1 Running 0 23h 10.200.1.27 k8worker2.workersubnet.testvcn.oraclevcn.com <none> <none>
kube-system coredns-6f57cbdd5b-mctlq 1/1 Running 0 3m59s 10.200.1.31 k8worker2.workersubnet.testvcn.oraclevcn.com <none> <none>
[opc@k8master1 ~]$ kubectl describe deployment/coredns -n=kube-system
Name: coredns
Namespace: kube-system
CreationTimestamp: Thu, 02 Jan 2025 13:28:10 +0000
Labels: app.kubernetes.io/instance=coredns
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=coredns
app.kubernetes.io/version=1.11.4
helm.sh/chart=coredns-1.37.0
k8s-app=coredns
kubernetes.io/cluster-service=true
kubernetes.io/name=CoreDNS
Annotations: deployment.kubernetes.io/revision: 4
meta.helm.sh/release-name: coredns
meta.helm.sh/release-namespace: kube-system
Selector: app.kubernetes.io/instance=coredns,app.kubernetes.io/name=coredns,k8s-app=coredns
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 1 max unavailable, 25% max surge
Pod Template:
Labels: app.kubernetes.io/instance=coredns
app.kubernetes.io/name=coredns
k8s-app=coredns
Annotations: checksum/config: 00f4dfa5bcd48c8950928182a87ad2f0b72a91ea458fd32e5dde421177e7c5ba
kubectl.kubernetes.io/restartedAt: 2025-01-03T12:44:31Z
scheduler.alpha.kubernetes.io/tolerations: [{"key":"CriticalAddonsOnly", "operator":"Exists"}]
Service Account: default
Containers:
coredns:
Image: coredns/coredns:1.11.4
Ports: 53/UDP, 53/TCP, 9153/TCP
Host Ports: 0/UDP, 0/TCP, 0/TCP
Args:
-conf
/etc/coredns/Corefile
Limits:
cpu: 100m
memory: 128Mi
Requests:
cpu: 100m
memory: 128Mi
Liveness: http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
Readiness: http-get http://:8181/ready delay=30s timeout=5s period=10s #success=1 #failure=5
Environment: <none>
Mounts:
/etc/coredns from config-volume (rw)
Volumes:
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: coredns
Optional: false
Node-Selectors: <none>
Tolerations: <none>
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: coredns-766cfd6db4 (0/0 replicas created), coredns-779d97db5 (0/0 replicas created), coredns-f7d5f96b4 (0/0 replicas created)
NewReplicaSet: coredns-6f57cbdd5b (1/1 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 36m (x3 over 85m) deployment-controller Scaled up replica set coredns-766cfd6db4 to 2 from 1
Normal ScalingReplicaSet 27m (x3 over 82m) deployment-controller Scaled down replica set coredns-766cfd6db4 to 1 from 2
Normal ScalingReplicaSet 26m deployment-controller Scaled up replica set coredns-766cfd6db4 to 1 from 0
Normal ScalingReplicaSet 23m (x2 over 26m) deployment-controller Scaled down replica set coredns-766cfd6db4 to 0 from 1
Normal ScalingReplicaSet 23m deployment-controller Scaled up replica set coredns-779d97db5 to 1
Normal ScalingReplicaSet 17m deployment-controller Scaled up replica set coredns-779d97db5 to 2 from 1
Normal ScalingReplicaSet 15m deployment-controller Scaled up replica set coredns-f7d5f96b4 to 1
Normal ScalingReplicaSet 15m deployment-controller Scaled down replica set coredns-779d97db5 to 1 from 2
Normal ScalingReplicaSet 15m deployment-controller Scaled up replica set coredns-f7d5f96b4 to 2 from 1
Normal ScalingReplicaSet 15m deployment-controller (combined from similar events): Scaled down replica set coredns-779d97db5 to 0 from 1
Normal ScalingReplicaSet 5m3s deployment-controller Scaled down replica set coredns-f7d5f96b4 to 1 from 2
Normal ScalingReplicaSet 4m27s deployment-controller Scaled up replica set coredns-6f57cbdd5b to 1
Normal ScalingReplicaSet 4m27s deployment-controller Scaled down replica set coredns-f7d5f96b4 to 0 from 1
[opc@k8master1 ~]$ kubectl describe svc/coredns -n=kube-system
Name: coredns
Namespace: kube-system
Labels: app.kubernetes.io/instance=coredns
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=coredns
helm.sh/chart=coredns-1.37.0
k8s-app=coredns
kubernetes.io/cluster-service=true
kubernetes.io/name=CoreDNS
Annotations: meta.helm.sh/release-name: coredns
meta.helm.sh/release-namespace: kube-system
Selector: app.kubernetes.io/instance=coredns,app.kubernetes.io/name=coredns,k8s-app=coredns
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.32.0.10
IPs: 10.32.0.10
Port: udp-53 53/UDP
TargetPort: 53/UDP
Endpoints: 10.200.1.31:53
Port: tcp-53 53/TCP
TargetPort: 53/TCP
Endpoints: 10.200.1.31:53
Session Affinity: None
Internal Traffic Policy: Cluster
Events: <none>
[opc@k8master1 ~]$
--------------------------------------------------------------------------------------------------------------------------------->
Scenario 1: With 1 POD under coredns Deployment
Observation: DNS lookup works fine when request is routed to coredns pod (10.200.1.31) via service (coredns/10.32.0.10)
[opc@k8master1 ~]$ kubectl exec -i -t dnsutils -- nslookup coredns.kube-system
Server: 10.32.0.10
Address: 10.32.0.10#53
Name: coredns.kube-system.svc.cluster.local
Address: 10.32.0.10
[opc@k8master1 ~]$ kubectl exec -i -t dnsutils -- nslookup kubernetes
Server: 10.32.0.10
Address: 10.32.0.10#53
Name: kubernetes.default.svc.cluster.local
Address: 10.32.0.1
[opc@k8master1 ~]$ kubectl exec -i -t dnsutils -- nslookup google.com
Server: 10.32.0.10
Address: 10.32.0.10#53
Non-authoritative answer:
Name: google.com
Address: 142.250.72.174
Name: google.com
Address: 2607:f8b0:4007:803::200e
[opc@k8master1 ~]$
Scenario 2: With 2 POD under coredns Deployment
Observation: DNS lookup is broken when DNS request is sent to new coredns POD (10.200.0.24) via service (coredns/10.32.0.10)
[opc@k8master1 ~]$ kubectl get pods -o wide --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
default dnsutils 1/1 Running 0 23h 10.200.0.17 k8worker1.workersubnet.testvcn.oraclevcn.com <none> <none>
default nginx 1/1 Running 0 23h 10.200.1.27 k8worker2.workersubnet.testvcn.oraclevcn.com <none> <none>
kube-system coredns-6f57cbdd5b-b2kxf 1/1 Running 0 109s 10.200.0.24 k8worker1.workersubnet.testvcn.oraclevcn.com <none> <none>
kube-system coredns-6f57cbdd5b-mctlq 1/1 Running 0 8m51s 10.200.1.31 k8worker2.workersubnet.testvcn.oraclevcn.com <none> <none>
[opc@k8master1 ~]$
[opc@k8master1 ~]$ kubectl exec -i -t dnsutils -- nslookup coredns.kube-system
Server: 10.32.0.10
Address: 10.32.0.10#53
Name: coredns.kube-system.svc.cluster.local
Address: 10.32.0.10
;; reply from unexpected source: 10.200.0.24#53, expected 10.32.0.10#53
;; reply from unexpected source: 10.200.0.24#53, expected 10.32.0.10#53
;; reply from unexpected source: 10.200.0.24#53, expected 10.32.0.10#53
;; connection timed out; no servers could be reached
[opc@k8master1 ~]$ kubectl exec -i -t dnsutils -- nslookup kubernetes
Server: 10.32.0.10
Address: 10.32.0.10#53
Name: kubernetes.default.svc.cluster.local
Address: 10.32.0.1
[opc@k8master1 ~]$ kubectl exec -i -t dnsutils -- nslookup google.com
;; reply from unexpected source: 10.200.0.24#53, expected 10.32.0.10#53
;; reply from unexpected source: 10.200.0.24#53, expected 10.32.0.10#53
;; reply from unexpected source: 10.200.0.24#53, expected 10.32.0.10#53
;; connection timed out; no servers could be reached
command terminated with exit code 1
[opc@k8master1 ~]$ kubectl exec -i -t dnsutils -- nslookup kubernetes
Server: 10.32.0.10
Address: 10.32.0.10#53
Name: kubernetes.default.svc.cluster.local
Address: 10.32.0.1
[opc@k8master1 ~]$ kubectl exec -i -t dnsutils -- nslookup kubernetes
Server: 10.32.0.10
Address: 10.32.0.10#53
Name: kubernetes.default.svc.cluster.local
Address: 10.32.0.1
[opc@k8master1 ~]$ kubectl exec -i -t dnsutils -- nslookup coredns.kube-system
;; reply from unexpected source: 10.200.0.24#53, expected 10.32.0.10#53
;; reply from unexpected source: 10.200.0.24#53, expected 10.32.0.10#53
^C
command terminated with exit code 1
[opc@k8master1 ~]$ kubectl exec -i -t dnsutils -- nslookup kubernetes
;; reply from unexpected source: 10.200.0.24#53, expected 10.32.0.10#53
^C
command terminated with exit code 1
[opc@k8master1 ~]$
