Hi Everyone, I have setup prometheus and grafana in a VM ( OnPrem ) and we do ha . . .

Observability And Monitoring
#1

Daemon:
Hi Everyone,
I have setup prometheus and grafana in a VM ( OnPrem ) and we do have multiple kubernetes clusters across AWS and Azure. we want to setup a centralized monitoring system. so for that purpose we had externalized prometheus and grafana in a VM.

Note: These multiple clusters includes multiple nodes (with both linux and windows)
My plan is to setup node exporter in cluster, expose the metrics and scrape them from prometheus which was setup externally. But what about windows nodes ? do i need to setup windows exporter as well ?

Can someone suggest an optimized way of proceeding ahead in this ?
Thanks!

1 Like
#2

Alex Novik:
you can use Prometheus federation feature
https://prometheus.io/docs/prometheus/latest/federation/

#3

Daemon:
yeah that’s fine. But how can i install exporters ? like as i said i do have both linux and windows nodes in a cluster.

#4

Alistair Mackay:
Yes, you need a node exporter for ALL nodes. Prometheus provides node exporters for both, but the metrics for windows and for linux are different because they are different operating systems.

Look on the grafana community dashboards to get a windows dashboard and a linux dashboard that work with the two exporters.

#5

Daemon:
you mean does node exporter works for both linux and windows nodes that are present in a cluster ?

#6

Alistair Mackay:
You install node exporter for windows on windows nodes, and node exporter for linux on linux nodes

#7

Alistair Mackay:
• https://github.com/prometheus-community/windows_exporter
• https://github.com/prometheus/node_exporter (linux)

#8

Daemon:
Thanks,

I’have one doubt. In my case: The cluster I’m working on has Autoscaling enabled. So, how can i dynamically scrape the metrics even when a new node joins to the cluster without manual intervention?

#9

Alistair Mackay:
You’ll need to set up service discovery. Due to the dynamic nature of kube, this is the only real option.

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config|https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config

#10

Alistair Mackay:
You’ll find many sorts of service discovery configurations in this section of the documentation

#11

Daemon:
Thanks! I’ll have a look on it

#12

Daemon:
Hey. I’ve gone through the service discovery. Looks like that doesn’t work when the prometheus is setup externally.
As I said we have setup promethues externally in a VM. I’ve configured the service discovery in .yml file but I got this error.

prometheus[4192031]: ts=2023-11-03T08:32:03.482Z caller=manager.go:317 level=error component="discovery manager scrape" msg="Cannot create service discovery" err="unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined" type=kubernetes config=kubernetes-nodes
#13

Alistair Mackay:
I haven’t done this, but it ought to work if you give Prometheus a kubeconfig file, just like what is used for kubectl.
“in-cluster configuration” is what the Kubernetes application SDK which Prometheus kubernetes-sd is almost certainly built on, will fall back to this mechanism if a kubeconfig can’t be found.

So try this. I assume you have a linux user account (e.g. prometheus) that the service runs as. Ensure this login has a home directory associated with it. Then in that home directory, create .kube directory and place a kubeconfig file in there. Use your own one that you use for kubectl initially to test, and if it works, then generate a kubeconfig specifically for Prometheus to use.

#14

Alistair Mackay:
Ensure also that there is a network route from the server Prometheus runs on to the cluster API endpoint.

#15

Daemon:
Sounds great. I’ll give it a try :wink:
Thanks!!

#16

Immanu தமிழன்:
@Daemon Were you able successfully discover the kubernetes nodes using kubernetes_sd_config plugin ?

#17

Daemon:
No Immanu. That isn’t working. I’ve to try as Alistair said in the last reply.

#18

Immanu தமிழன்:
already tried what has been said above , but still prometheus missing auto discovery

#19

Immanu தமிழன்:
started a new thread to see if any one has an idea on this https://kodekloud.slack.com/archives/C04ENBZ8TAM/p1699273882879009

#20

Alistair Mackay:
If you look beneath https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config, it gives an example Prometheus configuration where you point it to a kubeconfig file

# The information to access the Kubernetes API.

# The API server addresses. If left empty, Prometheus is assumed to run inside
# of the cluster and will discover API servers automatically and use the pod's
# CA certificate and bearer token file at /var/run/secrets/kubernetes.io/serviceaccount/.
[ api_server: <host> ]

# The Kubernetes role of entities that should be discovered.
# One of endpoints, endpointslice, service, pod, node, or ingress.
role: <string>

# Optional path to a kubeconfig file.
# Note that api_server and kube_config are mutually exclusive.
[ kubeconfig_file: <filename> ]

etc.