Network policy blocking all the ingress traffic

Kubernetes
#1

For the question ** One of the nginx based pod called cyan-pod-cka28-trb is running under cyan-ns-cka28-trb namespace and it is exposed within the cluster using cyan-svc-cka28-trb service.

This is a restricted pod so a network policy called cyan-np-cka28-trb has been created in the same namespace to apply some restrictions on this pod.

Two other pods called cyan-white-cka28-trb1 and cyan-black-cka28-trb are also running in the default namespace.

The nginx based app running on the cyan-pod-cka28-trb pod is exposed internally on the default nginx port (80 ).** → I created a network policy such that the detailed description is show as

np
np1058Ă—482 72.3 KB

I’m still unable to access the intended pod as show below
error
error1055Ă—180 37.1 KB

What mistake am I making

#2

HI @lavanya.singaravelan,

Please take in consideration the following :

  • cyan-white-cka28-trb1 and cyan-black-cka28-trb are running on the default namespace, but the network policy is created on namespace cyan-ns-cka28-trb.

  • We have a service to expose the pod cyan-pod-cka28-trb

Check this page to learn more on network policy

Regard

#3

I have used both the namespace selector with the labels of the default namespace and the pod selector

I referred this example(from the official documentation) while creating the above mentioned policy

...
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          user: alice
      podSelector:
        matchLabels:
          role: client
  ...

Since the pod on which the np has to applied is in cyan-ns-cka28-trb namespace, I created the np in the cyan-ns-cka28-trb namespace

#4

Hi @lavanya.singaravelan,

Normally, your ingress block should be something like :

ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: default
      podSelector:
        matchLabels:
          <!-- common label between the two pod -->
#5

Actually the name of the pod and the label is a bit different. The name of the pod is cyan-white-cka28-trb1 and the label is app=cyan-white-cka28-trb. I gave the label as app=cyan-white-cka28-trb1. That was the issue

#6

According to the question, it supposed to working from cyan-white-cka28-trb, when we use namespace and pod selector in a single array, not sure why the pod can be accessed from both white and black pods in default namespace

#7

I think problem here is that having ingress rule allowing access from pod cyan-white-cka28-trb1 from default namespace is not enough for “curl” test to work if you try accessing application via service name rather than pod IP.

I adjusted network policy to allow ingress to port 80 from pod cyan-white-cka28-trb1 and if I do curl from pod cyan-white-cka28-trb1 to IP of pod cyan-pod-cka28-trb it works. However, if I do curl using service name - it does not.

If I remove network policy - both curl requests work and if I check nginx log on cyan-ns-cka28-trb:

  • When curl is made to pod IP - request is coming from IP of pod cyan-white-cka28-trb1
  • When curl is made to service name - request is coming from IP of node that pod cyan-white-cka28-trb1 is on

So to make curl request work with service name, on top of having ingress rule for namespace+pod I had to add another one that allows access from IP addresses of cluster nodes.

Kodekloud team, I think it makes sense to double check this question and solution.

1 Like
#8

I’m have this same issue, it was driving me crazy. I’m glad it appears that it’s something the Kodekloud team may need to look at…

#9

I had same issue last day, I have tried f…g every thing. Even I don’t have podSelector just check with only namespaceSelector to at lease allow traffic from any pod from default namespace. Refer each and every part of ingress from official docs, Also I copy the solution from

but insanely this also didn’t work at all. When network policy removed traffic flows. This is from kodekloud side

#10

Hello @razabasit88

Please follow carefully my guide here which explains clearly how to solve it https://github.com/kodekloudhub/certified-kubernetes-administrator-course/blob/master/docs/16-Ultimate-Mocks/02-Troubleshooting/docs/19-C1-netpol-cyan-pod-cka28-trb.md

Everybody seems to have issues with this question!

#11

Your solution doesn’t work with the lab @Alistair_KodeKloud. Have you tested your solution on the actual lab?
I think the problem is what safonovi2000 described.

  • When curl is made to pod IP - request is coming from IP of pod cyan-white-cka28-trb1
  • When curl is made to service name - request is coming from IP of node that pod cyan-white-cka28-trb1 is on

And the CoreDNS is quite strange on the lab, sometimes it works, sometime it doesn’t

/home # nslookup cyan-svc-cka28-trb.cyan-ns-cka28-trb.svc.cluster.local
Server: 10.96.0.10
Address: 10.96.0.10:53

*** Can’t find cyan-svc-cka28-trb.cyan-ns-cka28-trb.svc.cluster.local: No answer

/home # nslookup cyan-svc-cka28-trb.cyan-ns-cka28-trb.svc.cluster.local
Server: 10.96.0.10
Address: 10.96.0.10:53

Name: cyan-svc-cka28-trb.cyan-ns-cka28-trb.svc.cluster.local
Address: 10.103.32.105

#12

Yes, many times @ks.trunghieu.95 ! Just about every time somebody posts about this question, which is 3-4 times a week! I’m somewhat offended that you think I would give a solution without testing it. By the number of posts about this, it is clearly the most difficult question in the entire mock series.

For your benefit I have tested it again right now. Note date stamp I have placed in the terminal

image
image1467Ă—623 104 KB

#13

I apologize for any offense, but I doubt that the lab I am using is identical to the one you got. In my opinion, the problem is the configuration of the Weave CNI, which includes the --cluster-cidr flag. This causes all requests from a pod to a service (point to pod on another node) to be NATed, and the source IP is changed to the node IP instead of the pod IP. With this source IP, network policies will not work.

#14

Is this repeatable? I have not seen it in the twenty or so times I’ve done this question since creating the solution.

#15

OK, I got the lab after ton of click. Thank for the random.
On the lab there are two worker node: node01 and node02
The IPALLOC_RANGE on weave set to 10.50.0.0/16
The clusterCIDR on kube-proxy set to: 10.244.0.0/16
(the note from weave doc Integrating Kubernetes via the Addon state that: " * If you do set the --cluster-cidr option on kube-proxy, make sure it matches the IPALLOC_RANGE given to Weave Net" )

If the pod of white or black stay on the same node with cyan-pod-cka28-trb SNAT not happen and thing works.
But if they stay on different node it will SNAT packet to change ip source to Node IP which we can see on the iptables rule:
-A KUBE-SVC-A53YBJOHBLWETSHT ! -s 10.244.0.0/16 -d 10.100.64.59/32 -p tcp -m comment --comment “cyan-ns-cka28-trb/cyan-svc-cka28-trb cluster IP” -m tcp --dport 80 -j KUBE-MARK-MASQ

#16

Sorry for my English. Here is the Bard version:

Analysis of the lab environment

The lab environment has two worker nodes, node01 and node02, and the IPALLOC_RANGE on Weave is set to 10.50.0.0/16. The clusterCIDR on kube-proxy is set to 10.244.0.0/16. This is a mismatch, as the clusterCIDR must match the IPALLOC_RANGE given to Weave Net.

Impact of the mismatch on SNAT

If a pod of white or black stays on the same node as the cyan-pod-cka28-trb, SNAT does not happen. However, if the pods are on different nodes, SNAT will happen and the source IP will be changed to the node IP. This can be seen in the iptables rule:

-A KUBE-SVC-A53YBJOHBLWETSHT ! -s 10.244.0.0/16 -d 10.100.64.59/32 -p tcp -m comment --comment “cyan-ns-cka28-trb/cyan-svc-cka28-trb cluster IP” -m tcp --dport 80 -j KUBE-MARK-MASQ

This rule matches all TCP packets on port 80 that are not from the 10.244.0.0/16 network and are destined for the 10.100.64.59/32 network. The KUBE-MARK-MASQ target marks the packets for SNAT, which means that the source IP will be changed to the node IP.

Impact of SNAT on network policies

If SNAT happens, network policies will not work as expected. This is because network policies are based on pod IPs, not node IPs. If a pod’s IP is changed to the node IP, the network policy will no longer be able to match the pod.

Recommendations

To fix the problem, you need to make sure that the clusterCIDR on kube-proxy matches the IPALLOC_RANGE given to Weave Net. Once you have done this, SNAT will no longer happen and network policies will work as expected.

Here are the steps to fix the problem:

  1. Edit the kube-proxy configuration file and set the cluster-cidr flag to match the IPALLOC_RANGE given to Weave Net.
  2. Restart kube-proxy.
    kubectl rollout restart daemonset -n kube-system kube-proxy
#17

Nice investigation :+1: I retract all offense :slight_smile:

Clearly I have had the luck not to have the pods scheduled on separate nodes in all the times I have run this lab.

I shall make a reproduction of this by manually scheduling pods and get it looked at by the lab team.

#18

Very interesting @ks.trunghieu.95

I’ve just run up the lab again and this is what I get

student-node ~ âžś  k exec -n kube-system weave-net-7dx2p -c weave -- printenv | grep IPALLOC
IPALLOC_RANGE=10.244.0.0/16

and

student-node ~ âžś  k get cm -n kube-system kube-proxy -o jsonpath={'.data.config\.conf}' | yq e .clusterCIDR -
10.244.0.0/16

Clearly we are looking at an intermittent bug in the lab setup. Certainly other students have got a pass from the given solution - this thread was marked as solution by someone before you added to it.

I will create an internal ticket and reference this thread.

#19

these IP ranges are supposed to match right? in my lab for this question I ran both commands but they showed different IP ranges on each. I looked at the 3 pods and they were all scheduled on the same node with same IP range. it was the removal of the dash in front of podSelector that got the policy to work for me.

#20

As pointed out, if the IP ranges are different, then it’s a lab setup bug and the policy only works when the pods are all scheduled on the same node.