Self healing infrastructure with prometheus, ansible, alertmanager

DevOps
#1

I’m doing an internship. And today i got a task to create a self healing infrastructure with prometheus, ansible, alertmanager.As I didn’t covered these topics so i used chatgpt to help me with this task.
Then, i created project folder and 4 containers (nginx, prometheus, alertmanager, webhook)
folder struccture-

webhook/

  • app.py
  • paybook.yml
  • Dockerfile
    alertmanager.yml
    docker-compose.yml
    prometheus.yml
    rules.yml

All containers are running. To test this process. When i stopped nginx manually then Prometheus alert is in firing state and send alert to alertmanager. but the problem is that , the nginx xontainer not starting automatically by playbook…
It’s urgent.

#2

Hi @Rajsingh,

I’m confused about why the Nginx container isn’t restarting automatically with the Ansible playbook. From what I understand, you’re using Docker Compose to deploy the containers. If you stop a container manually, you’ll need to check the Docker Compose file to make sure the auto-restart setting is enabled, instead of relying on the Ansible playbook.

1 Like
#3

Screenshot_2025-04-22-22-46-01-87_e2d5b3f32b79de1d45acd1fad96fbb0f
Screenshot_2025-04-22-22-46-01-87_e2d5b3f32b79de1d45acd1fad96fbb0f1080×472 89.5 KB

Can you help me with this project?

#4

The key to this is the webhook receiver. @raymond.baoly - using compose auto-start is not in the spirit of this task, which is to use alerts from alert manager to perform mitigation actions.

@Rajsingh where does the python code for the webhook receiver come from? Were you given it or did you code it yourself?

The job of the webhook receiver is to decode the JSON payload received from alert manager and then perform an action based on that data, i.e. execute the playbook. So:

  • Assert prometheus is detecting the down condition (in Prometheus UI)
  • Assert that your alert manager rules are correct and an alert is being fired (in alert manager UI)
  • Assert that the webhook is running and listening. The webhook should write a log entry when it receives something so you can determine it is actually getting messages from alert manager. To test the webhook in isolation, create a dummy JSON alert payload and send it to the webhook using curl.
  • Assert that the routing rules are correct for the alert such that the webhook will be invoked when the alert is raised.
#5

Hey, I resolved this problem. Actual problem was that the docker-cli was not installed on webhook container so the ansible was not able to create the nginx container. So i had to install docker-cli on webhook container.
@raymond.baoly @Alistair_KodeKloud