Need some clarification on authorization header for k8s service discovery i.e kubernetes_sd_configs?
Question 1:
why we need authorization header for the role endpoints i.e (- role: endpoints) but not pods i.e (- role: pod)?
More ref: prometheus/documentation/examples/prometheus-kubernetes.yml at release-2.54 · prometheus/prometheus · GitHub
Question 2:
As per prometheus doc, The discovery auth config is automatic if Prometheus runs inside the cluster. Otherwise, more config options have to be provided within the <kubernetes_sd_config?
That means it will use service account attached to prometheus pod for authentication with k8s api server?
Judging by the content of that config.yaml file linked from Github, Prometheus is using the Kubernetes API to communicate with K8s. Over the wire, this is using an Authorization header in an HTTP request. This uses a token that’s exposed in containers at the path /var/run/secrets/kubernetes.io/serviceaccount/token, which is the standard location K8s uses for service accounts bound to a pod. So if Prometheus is running inside the cluster, these are the standard locations, and it won’t take special configuration to get them.
If prometheus is not running inside the cluster, it will need the same information – a token and the appropriate CA to authenticate the cluster to Prometheus, but you’ll have to supply them some other way.
yeah even that was my understanding, thanks rob for clarification.
But the confusion comes due to below statement from doc. Then we why need to provide auth again for each job type. For ex: - job_name: “kubernetes-apiservers”
This TLS & authorization config is used to connect to the actual scrape
# endpoints for cluster components. This is separate to discovery auth
# configuration because discovery & scraping are two separate concerns in
# Prometheus. The discovery auth config is automatic if Prometheus runs inside
# the cluster. Otherwise, more config options have to be provided within the
# <kubernetes_sd_config>.