Hi all, I am trying to access the kubeapi server from one of worker nodes using . . .

Kubernetes
#1

Krishna Yeddula:
Hi all, I am trying to access the kubeapi server from one of worker nodes using the curl command
curl https://172.31.39.212:6443/api/v1/namespaces|https://master:6443/api/v1/namespaces --cert kubelet-client-current.pem --key kubelet-client-current.pem --cacert /etc/kubernetes/pki/ca.crt -k
but getting an error

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "namespaces is forbidden: User \"system:anonymous\" cannot list resource \"namespaces\" in API group \"\" at the cluster scope",
  "reason": "Forbidden",
  "details": {
    "kind": "namespaces"
  },
  "code": 403

I am not sure why it is saying the client belongs to system:anonymous as I checked the pem file which is showing this belongs to system:nodes

[root@k8s-worker2 pki]#openssl x509 -in kubelet-client-current.pem -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            []
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
        Validity
            Not Before: Oct 23 19:12:09 2021 GMT
            Not After : Oct 23 19:12:09 2022 GMT
        Subject: O=system:nodes, CN=system:node:k8s-worker2
        Subject Public Key Info:

    and the /etc/kubernetes/kubelet.conf is listing
   client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
   client-key: /var/lib/kubelet/pki/kubelet-client-current.pem
#2

Lukasz Matuszczak:
Hi,
Something is wrong here. In your example both “cert” and “key” refer to the same file. It seems that kubelet-client-current.pem is a certificate, because you can parse it with openssl x509. The key should be in separate file.

#3

Krishna Yeddula:
@Lukasz Matuszczak Thanks for the response. I installed the cluster with kubeadm so these are the out-of -the-box configs

#4

Lukasz Matuszczak:
You are right. Pem file may contain both certificate and key. At first glance, I cannot see any mistake in your curl command. I will check it later on my kubeadm-managed cluster.

#5

Krishna Yeddula:
Sure @Lukasz Matuszczak looking forward to your updates and do let me know the client-side auth (initiating a connection from kubelet to kube-api server) is working in your case

curl <https://master:6443/api/v1/namespaces> --cert &lt;kubelet-client-cert&gt; --key &lt;kubelet-client-key&gt; --cacert &lt;ca-cert&gt; -k
#6

Lukasz Matuszczak:
Hi, I checked the Kubernetes 1.22.3 installed on machines prepared using the Vagrant setup from https://github.com/kodekloudhub/certified-kubernetes-administrator-course and Kubeadm: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/.

Your example almost works. The first call failed but because of a different error:

cd /var/lib/kubelet/pki
curl <https://192.168.56.2:6443/api/v1/namespaces> --cert kubelet-cliet-current.pem --key kubelet-client-current.pem --cacert /etc/kubernetes/pki/ca.crt -k

{                                                                                                                                                          
  "kind": "Status",                                                                                                                                        
  "apiVersion": "v1",                                                                                                                                      
  "metadata": {                                                                                                                                            
                                                                                                                                                           
  },                                                                                                                                                       
  "status": "Failure",                                                                                                                                     
  "message": "namespaces is forbidden: User \"system:node:kubenode01\" cannot list resource \"namespaces\" in API group \"\" at the cluster scope",        
  "reason": "Forbidden",                                                                                                                                   
  "details": {                                                                                                                                             
    "kind": "namespaces"                                                                                                                                   
  },                                                                                                                                                       
  "code": 403                                                                                                                                              
}

The system:node ClusterRole is not allowed to list namespaces, but for example if I replace it with v1/nodes in curl command it works perfectly.

I use the IP address of the master node, not master DNS name. This was the default configuration generated by kubeadm in: /etc/kubernetes/kubelet.conf

Maybe in your case master refers to some other cluster API server? If Kubernetes API Server could not recognize the user, then the default is system:anonymous . So check if the node k8s-worker2 really exists in the cluster by running kubectl get nodes.