I need one help. Right now I working on Prometheus Storage config section. This . . .

Asokan Chinnasamy:
I need one help.
Right now I working on Prometheus Storage config section.
This is my prometheus config
- name: prometheus-server
image: “prom/prometheus:v2.28.1”
imagePullPolicy: “IfNotPresent”
args:
- --storage.tsdb.retention.time=15d
- --config.file=/etc/config/prometheus.yml
- --storage.tsdb.path=/data
- --web.console.libraries=/etc/prometheus/console_libraries
- --web.console.templates=/etc/prometheus/consoles
- --web.enable-lifecycle
- --storage.tsdb.wal-compression

Issue is that /data space is rapidly increasing.
I given 20GB space(pvc) to mount on /data.
within 24Hrs 20GB space is filled
This config taken from istio addon only
What I want to do for space utilization?

unnivkn:
You have to restrict your logs, by means of filtering only the most wanted & remove everything else. Also make sure a cleanup strategy, to remove old files from your disk.

unnivkn:
https://github.com/prometheus/prometheus/issues/3487