Hello <#C04ENBZ8TAM|observability> team, Can anybody help with Prometheus query. . . .

Observability And Monitoring
#1

Vinod Patil:
Hello <#C04ENBZ8TAM|observability> team, Can anybody help with Prometheus query. My requirement is that Prometheus and targets will remain in customer network. I want to replicate that Prometheus data to my Org’s network and connect Grafana to replicated data for visualization.

  1. I am going through “remote write” feature but will it be useful in this case.
  2. Can the remote write be another Prometheus server itself ? ( instead of other supported remote storage options )
#2

Conall Ó Gribín:
1] Yes

2] Yes - enable the remote write receiver feature via the command line flag --web.enable-remote-write-receiver

https://prometheus.io/docs/prometheus/latest/storage/#overview

once enabled the api/v1/write endpoint will be exposed.

#3

Conall Ó Gribín:
If you plan to scale this out so you have multiple customer prom instances, and a centralised prom instance, a more scaleable solution would be to use a prom compatible backend like https://thanos.io/|Thanos, https://cortexmetrics.io/docs/getting-started/|Cortex, or https://grafana.com/oss/mimir/|Mimir.

#4

Vinod Patil:
@Conall Ó Gribín Thanks for the answer, that really helped. I was also thinking to have external backend for scalability. Because there are multiple customers.

#5

Vinod Patil:
@Conall Ó Gribín Is there a way to store the data separately for each customer ? Because we want customer specific visualization.

#6

Conall Ó Gribín:
Grafana Mimir, Thanos and Cortex all support the concept of tenants. This is your method to seperate customer data. See the documentation for whatever solution you decide to choose.

#7

Conall Ó Gribín:
e.g for Mimir: https://grafana.com/docs/mimir/latest/manage/secure/authentication-and-authorization/

and Thanos: https://thanos.io/tip/operating/multi-tenancy.md/

#8

Conall Ó Gribín:
An example with Thanos is here: https://www.infracloud.io/blogs/multi-tenancy-monitoring-thanos-receiver/

#9

Vinod Patil:
@Conall Ó Gribín Thanks a lot for the response. I will look into it.

#10

Conall Ó Gribín:
V welcome :+1: