Data visualization on Grafana dashboard

Kubernetes
#1

I have an application that runs on Kubernetes to measure storage performance. Upon completion of each POD, it displays statistics in terms of IOPS, Bandwidth, and Latency.

Now, the requirement is to run the application daily and view the statistics on a Grafana dashboard. Can we send the data directly to Prometheus and view it on Grafana, or do we need any kind of database as a data source? Please assist me with this.

Please find the sample data here.

RANDOM_IOPS_READ,RANDOM_IOPS_WRITE,MIX_IOPS_READ,MIX_IOPS_WRITE,RANDOM_BW_READ,RANDOM_BW_WRITE,SEQUENSE_BW_READ,SEQUENCE_BW_WRITE,AVG_LATENCY_READ,AVG_LATENCY_WRITE
[67400.0, 8118.0, 1864.5, 613.5, 263.0, 31.8, 258.0, 0.93, 1.899, 15.782]
[82097.0, 9332.5, 1572.0, 527.0, 321.5, 37.25, 830.5, 5.721, 31.165, 271.93]
[78677.5, 8393.5, 1460.0, 476.5, 309.0, 33.9, 812.5, 7.262, 64.902, 606.7]
#2

If you format your data as a prometheus format file, then you can scrape the file content directly with file_sd_configs.

This means that instead of creating an excel style data table, you need to think in terms of what this data might look like as a metric.

So if these numbers are discrete totals generated at the end of a run, then each number is itself a metric of type Counter. I’m presuming each row is a pod.

In that case, each column name is a metric name and the metrics file would look something like this. You can add other labels if you have metadata other than the pod id/name or whatever you use to identify the pod.

# HELP pod_random_iops_read The total IOPS on random read.
# TYPE pod_random_iops_read counter
pod_random_iops_read{pod="pod1"} 67400.0
pod_random_iops_read{pod="pod2"} 82097.0
pod_random_iops_read{pod="pod3"} 78677.5

# HELP pod_random_iops_write The total IOPS on random write.
# TYPE pod_random_iops_write counter
pod_random_iops_write{pod="pod1"} 8118.0
pod_random_iops_write{pod="pod2"} 9332.5
pod_random_iops_write{pod="pod3"} 8393.5

…etc.

Prometheus will ingest this directly and you can display it in grafana.

#3

Hi @Alistair_KodeKloud

Thank you very much for your response. Could you please provide your suggestions on how to ingest these metrics by using Python, as my applications are written in Python? Thanks again for your help.

#4

Python does not ingest the metrics, Prometheus does.

Your python script needs to create a file in the format I have given above, instead of in the format you posted in the first message.

You then write that file in a location accessible to Prometheus and set it up to scrape your file using the info in the links above.

If the python program is running continuously and not just once at end of day, then it can export metrics in real time by using the Prometheus metrics client in the application. It will serve an HTTP endpoint that Prometheus can connect to.

#5

Thank You so much @Alistair_KodeKloud . Your inputs hepled me a lot. I will make the changes.