I have two ingress setup on my Kubernetes cluster one for internal traffic called nginx-ingress and it has default setting and with values of
root@k8s-master-1:~/Ingress# cat values.yaml
controller:
name: ingress-nginx
ingressClassResource:
name: nginx
enabled: true
default: false
controllerValue: "k8s.io/ingress-nginx"
ingressClass: nginx
replicaCount: 1
service:
type: LoadBalancer
loadBalancerIP: 10.111.111.74
externalTrafficPolicy: Local
resources:
requests:
cpu: 500m
memory: 512Mi
limits:
cpu: 2
memory: 2Gi
metrics:
enabled: true
serviceMonitor:
enabled: false
# nodeSelector:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
topologyKey: "kubernetes.io/hostname"
labelSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values:
- ingress-nginx
I used helm charts and help create a ingress class for it. Then I create another ingress control with different name with following values.
root@k8s-master-1:~/Ingress# cat values-external.yaml
controller:
name: ext-ingress
ingressClassResource:
name: external
enabled: true
default: false
controllerValue: "k8s.io/ext-ingress"
ingressClass: external # Diffrent class name
replicaCount: 1
service:
type: LoadBalancer
loadBalancerIP: 10.111.111.75 # MetalLB external LB IP
externalTrafficPolicy: Local
metrics:
enabled: true
serviceMonitor:
enabled: false
resources:
requests:
cpu: "500m"
memory: "512Mi"
limits:
cpu: "2"
memory: "2Gi"
autoscaling:
enabled: true
minReplicas: 1
maxReplicas: 5
targetCPUUtilizationPercentage: 60
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
topologyKey: "kubernetes.io/hostname"
labelSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values:
- ext-ingress
I have to create a class for it as it was not default ingress and I create it as follow.
root@k8s-master-1:~/Ingress# cat ingressclass-external.yaml
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
name: external
annotations:
ingressclass.kubernetes.io/is-default-class: "false"
spec:
controller: k8s.io/ext-ingress
Problem:
I have 4 applications 3 of them should pass through internal ingress and one should pass through external ingress. One application is internal-portal and to login it with it I need to use Keycloak. When I add my username and password of KC it access and then my application keep surfing and I see error 401 in web console blinking. Finally I see a message site cant be reach.
The very strange behaviour is when I add Keycloak to use external class it works but if I search kubectl get ingress -A I can see my Keycloak is using external class but its IP is still internal class
root@k8s-master-1:~/Ingress# kubectl get ingress -A
NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE
api-uat gateway-api-external-ingress external gateway-api-uat 10.111.111.75 80, 443 21h
api-uat payment-api-ingress nginx payments-uat 10.111.111.74 80, 443 34d
ui-external-uat rtd-ext-portal-ingress nginx portal-uat 10.111.111.74 80, 443 35d
ui-internal-uat rtd-int-portal-ingress external int-portal-uat 10.111.111.75 80, 443 37d
utility-service-uat keycloak-uat-ingress external identity-uat 10.111.111.74 80, 443 34d
It is very strange as now KC and my int-portal are in different ingress and they work. I can login to int-portal by using KC and verification complete and I can see my application and use it. How does it work thats a big Question.
But when I add KC into same ingress it stop working. I can login to KC directly but whrn I use internal-portal and it ask my KC user and password after that it keep surfing and then site cant be reached.
I look into logs, noting found. I try changing different class and test still the same issue. I am not sure how can it has different class and different IP. I changed the values to specify use controller name to pick IP as you can see in my value. Yaml. Non of my controller set as default so noting can be picked by default as well.
I want that we KC and other related app should be under one ingress and that should be work fine. I am not allow to change IP for KC/Internal ingress and KC should be under internal Ingress.