Cortex with multiple Prometheus instances

Hi all,
Any body using Cortex to store multiple Prometheus instances data ? I have configured 2 Prometheus instances which remote write to Cortex. I am getting error in Prometheus which I am pasting , similar error is there in Cortex log.

ts=2024-01-09T13:37:19.593Z caller=dedupe.go:112 component=remote level=error remote_name=bdd0e8 url= msg="non-recoverable error" count=500 exemplarCount=0 err="server returned HTTP status 400 Bad Request: maxFailure (quorum) on a given error family, rpc error: code = Code(400) desc = addr= state=ACTIVE zone=, rpc error: code = Code(400) desc = user=fake: err: out of order sample.

Check that the system time is in sync on all prometheus and cortex hosts - they should all be using the same NTP time source. If it’s not that, then I don’t know because I haven’t worked with cortex.

Thanks , I see that my both Prometheus instances are 1 hour behind than Cortex instance. It could be the issue. BTW I got this article which I am sharing here which may help someone who is facing similar errors.

Understanding Duplicate Samples and Out-of-order Timestamp Errors in Prometheus

I will try to sync the time Or will start Cortex in the same time zone and will put my observation here.

Not sure if the time stamp has to match but I was getting same error even when I tried on same timestamp. Then I tried changing the job name of each Prometheus instance.
e.g. job_name: “prometheus1” for first Prometheus instance and
job_name: “prometheus2” for second. After this, I am not getting out of order sample errors.

external labels are very much important if you are pushing data from multiple Prometheus instances to Cortex ( or Thanos etc )