Hi all,
Any body using Cortex to store multiple Prometheus instances data ? I have configured 2 Prometheus instances which remote write to Cortex. I am getting error in Prometheus which I am pasting , similar error is there in Cortex log.
ts=2024-01-09T13:37:19.593Z caller=dedupe.go:112 component=remote level=error remote_name=bdd0e8 url=http://150.132.197.86:9009/api/v1/push msg="non-recoverable error" count=500 exemplarCount=0 err="server returned HTTP status 400 Bad Request: maxFailure (quorum) on a given error family, rpc error: code = Code(400) desc = addr=150.132.197.86:9095 state=ACTIVE zone=, rpc error: code = Code(400) desc = user=fake: err: out of order sample.
Check that the system time is in sync on all prometheus and cortex hosts - they should all be using the same NTP time source. If it’s not that, then I don’t know because I haven’t worked with cortex.
Thanks , I see that my both Prometheus instances are 1 hour behind than Cortex instance. It could be the issue. BTW I got this article which I am sharing here which may help someone who is facing similar errors.
I will try to sync the time Or will start Cortex in the same time zone and will put my observation here.
Not sure if the time stamp has to match but I was getting same error even when I tried on same timestamp. Then I tried changing the job name of each Prometheus instance.
e.g. job_name: “prometheus1” for first Prometheus instance and
job_name: “prometheus2” for second. After this, I am not getting out of order sample errors.