Prometheus monitoring

Observability And Monitoring
#1

Dear people,
I installed minikube in ubuntu desktop and deployed an application with the matchlabels role:demo. Then, i installed prometheus as binary and used the below config to monitor the pods.

global:
scrape_interval: 15s

scrape_configs:

  • job_name: “demo”
    kubernetes_sd_configs:

    • api_server: ‘https://192.168.58.2:8443’
      role: pod
      #kubeconfig: ‘/tmp/config’
      bearer_token_file: ‘/root/.kube/config’

    tls_config:
    ca_file: ‘/root/.minikube/ca.crt’
    insecure_skip_verify: true
    relabel_configs:

    • source_labels: [__meta_kubernetes_namespace]
      action: keep
      regex: default # Specify namespace(s) to monitor

    • source_labels: [__meta_kubernetes_pod_label_role]
      action: keep
      regex: demo

But I am getting the error couldn’t list the pods in the cluster.
ts=2024-02-13T12:06:52.847Z caller=klog.go:108 level=warn component=k8s_client_runtime func=Warningf msg=“pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169: failed to list *v1.Pod: pods is forbidden: User "system:anonymous" cannot list resource "pods" in API group "" at the cluster scope”
ts=2024-02-13T12:06:52.848Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg=“pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:anonymous" cannot list resource "pods" in API group "" at the cluster scope”

I can able to list the pods using the config file mentioned in the config file.
Any help is appreciated.

#2

You have commented out the kubeconfig variable and have the bearer_token_file pointing to the kubeconfig.

Do you want to use the kubeconfig or the bearer token?

#3

I want to use the config file and when I tried using that, below error occured.
Checking prometheus.yaml
FAILED: parsing YAML file prometheus.yaml: yaml: unmarshal errors:
line 13: field kubeconfig not found in type kubernetes.plain

Tried using the kubeconfig key with different types and indentations. But still no luck.
Resolving this will also help me.

#4

I want to use the config file and when I tried using that, below error occured.
Checking prometheus.yaml
FAILED: parsing YAML file prometheus.yaml: yaml: unmarshal errors:
line 13: field kubeconfig not found in type kubernetes.plain

Tried using the kubeconfig key with different types and indentations. But still no luck.
Resolving this will also help me.